bpf: Support aggregate return values up to 16 bytes - #8695

LLVM 23 added support for returning a value in two registers for an
__int128, or a struct/union whose size is greater than 8 but not more than
16 bytes: such a value comes back in the R0:R2 register pair, with R2
holding the upper half. See LLVM patches [1] and [2].

Later patches teach the JIT, precision backtracking, live register analysis
and the verifier itself about that convention. All of them need to answer
the same question: does this subprogram return its value in a register
pair? Add the shared helpers up front so that those patches can be ordered
independently of each other:

 - subprog_ret_type() resolves a subprogram's BTF return type. It is
   factored out of subprog_returns_void(). The verifier_bug_if(!func) and
   !func_proto checks it replaces are redundant, since
   check_btf_func_early() already rejects a func_info record whose type_id
   is not a BTF_KIND_FUNC pointing at a BTF_KIND_FUNC_PROTO. A check on
   prog->aux->{btf,func_info} is added instead: unlike
   subprog_returns_void(), which is only used for global subprograms, later
   callers ask about static subprograms too, and those may belong to a
   program loaded without BTF.

 - ret_regs_cnt() maps the size of a return value to the number of
   registers holding it.

 - bpf_ret_reg_pair() answers the question above. Its users query it at
   every subprogram call and at every subprogram exit, that is once per
   verifier state rather than once per subprogram, so the answer is
   precomputed into bpf_subprog_info->ret_reg_pair by
   bpf_compute_subprog_ret_regs() and the helper itself is a flag test.
   It lives in bpf_verifier.h because kernel/bpf/backtrack.c and
   kernel/bpf/liveness.c need it as well.

bpf_compute_subprog_ret_regs() runs in bpf_check() right before
bpf_compute_live_registers(), which is the first of those users: by then
BTF func_info has been validated and the subprogram list is final.

No functional change, bpf_ret_reg_pair() has no callers yet.

  [1] llvm/llvm-project#190894
  [2] llvm/llvm-project#206876

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>

LLVM 23 returns an __int128, or a struct/union larger than 8 bytes and no
larger than 16 bytes, in the BPF R0:R2 register pair. The previous patch
taught the verifier about that convention; wire up the JIT side so that the
second half of the return value actually lands in R2.

A kfunc returning more than 8 bytes hands the second half of the result
back in RDX, the native x86-64 ABI's second return register. BPF R0 maps to
RAX so it needs no move, but BPF R2 maps to RSI, so emit a RDX->RSI move
after a BPF_PSEUDO_KFUNC_CALL whose function model reports ret_size > 8.

Placing the second return half into R2 is possible on any JIT, but it needs
architecture-specific JIT work. Rather than requiring every JIT to
implement it at once, add a bpf_jit_supports_kfunc_ret_reg_pair()
capability, defaulting to false in the generic core; an architecture opts
in once its JIT handles the R0:R2 pair, and the remaining ones are left for
future work. The verifier enforces it in bpf_add_kfunc_call(), rejecting a
kfunc whose return is larger than 8 bytes with -EOPNOTSUPP when the JIT
lacks the capability. Only x86, arm64 and riscv are supported so far.

On arm64 and riscv the native second return register is already BPF R2 (x1
in bpf2a64[] and a1 in regmap[] respectively), so the value is in the R0:R2
register pair on return with no extra move, unlike x86 (RDX->RSI). This has
been tested on x86 and arm64. The riscv path is expected to work by the
same register-mapping reasoning as arm64 but has not been tested.

bpf_add_kfunc_call() also rejects a kfunc that is marked KF_FASTCALL and
returns more than 8 bytes. The bpf_fastcall contract implemented by
mark_fastcall_pattern_for_call() assumes a call clobbers R0 plus the
registers holding its arguments, so a return in the R0:R2 pair would
clobber an R2 the caller expects the fastcall pattern to preserve. Such
a kfunc is rejected with -EOPNOTSUPP as well.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>

A function returning a value larger than 8 bytes (a struct/union, or an
__int128) uses R2 as a second return register alongside R0. Precision
backtracking treats only R0 as a return register at a call/return boundary,
so once the verifier starts modeling R2 that way, marking the second half
of such a return precise would trip the "unexpected regs" checks in
backtrack_insn() and reject a valid program with -EFAULT. Handle it here,
ahead of the patch that introduces the modeling.

Marking the upper half precise, for example by branching on it after a
call to a static subprogram, walks backtracking into the callee and
reaches its BPF_EXIT with R2 still set in the mask. R2 is part of
BPF_REGMASK_ARGS, so this hits "backtracking exit unexpected regs".
Returning the pair from a global subprogram or from a kfunc instead hits
the equivalent check at the call site.

Handle R2 like R0 in the three boundaries where a call defines the return
registers:

 - static subprog exit (BPF_EXIT): when the callee returns a pair, R2 is a
   return register rather than a clobbered argument, so its precision has to
   cross the frame boundary just like R0's: clear it from the caller's mask
   before the R1-R5 check, then set it again in the callee's mask after
   bt_subprog_enter().

   The clear has to be conditional, which is why the subprogram containing
   the exit insn is looked up and queried. For a callee that does not return
   a pair, check_func_call() has already invalidated the caller's R1-R5 and
   prepare_func_exit() copies back only R0, so nothing after the call can
   depend on R2 and backtracking should never still be asking for it here.
   Clearing it unconditionally would turn that into a silent no-op instead
   of reporting it through the existing "backtracking exit unexpected regs"
   check.
 - global subprog call: a global subprog returning >8 bytes also sets R2;
   clear it before the args check.
 - kfunc call (BPF_CALL): a kfunc returning >8 bytes (model ret_size > 8)
   also sets R2; clear it like R0.

All three are gated on R2 actually being in the mask, so the extra BTF and
kfunc descriptor lookups stay off the common backtracking path.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>

A BPF_EXIT of a subprogram returning a value larger than 8 bytes (a
struct/union or an __int128) reads R2 as well as R0, since the second half
of the return value is passed back in R2. compute_insn_live_regs() only
marked R0 used at exit, so a callee's R2 could be considered dead and
cleaned from checkpointed states, which would allow unsound state pruning.

Mark R2 as read at the BPF_EXIT of a subprogram that does return a register
pair. bpf_compute_live_registers() walks the instructions in order and
env->subprog_info[] is sorted by subprogram start, so the containing
subprogram is tracked with a running index and its return convention is
queried once per subprogram through bpf_ret_reg_pair().

Marking R2 at every exit instead would be simpler, but R2 would then stay
live backwards across any call that is not followed by a write to R2, which
is nearly every program, and would needlessly hurt state pruning.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>

A callback handed to a helper or a kfunc (bpf_loop(),
bpf_timer_set_callback(), bpf_for_each_map_elem(), ...) is invoked
through bpf_callback_t, and an exception callback is invoked by
bpf_throw() through

  u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);

Both prototypes yield a single u64 in R0, and neither caller has any
notion of a second return register, so a callback returning a value in
the R0:R2 pair would have the upper half of its return value silently
dropped.

Reject both at load time:

 - check_ld_imm(): a callback is materialized as PTR_TO_FUNC by an
   ld_imm64 pointing at its subprogram, so the subprogram's return
   convention can be checked where the callback pointer is created,
   before it ever reaches a helper or kfunc argument.

 - do_check_common(): an exception callback is not referenced by a
   PTR_TO_FUNC, it is named by a BTF decl_tag and verified on its own,
   so check it as its frame is set up, next to the existing "cannot
   return void" and single-argument checks.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>

LLVM 23 added support for returning a value in two registers for an
__int128, or a struct/union whose size is greater than 8 but not more than
16 bytes. See LLVM patches [1] and [2].

Before LLVM 23 the BPF backend could not return these values at all. A
by-value struct or union return (of any size) was rejected at compile time
with:

  error: aggregate returns are not supported

and an __int128 return failed later in the backend with:

  fatal error: error in backend: unable to allocate function return #1

Both are resolved in LLVM 23, which lowers such returns into the R0:R2
register pair.

This patch adds handling for returns greater than 8 bytes in several
places: BPF subprogram returns (the main program, and both global and
static subprograms) and kfunc returns.

The R0:R2 convention is only implemented in the JIT. The BPF interpreter
has no notion of a second return register: a BPF-to-BPF call goes through
JMP_CALL_ARGS and a BPF_EXIT hands back BPF_R0 alone, so a caller reading
R2 would see a stale value. Force the JIT wherever a caller can observe the
pair, that is at the call to a global subprogram in check_func_call() and
at the return from a static subprogram in prepare_func_exit(). Kfunc calls
need no separate handling since bpf_add_kfunc_call() already sets
jit_required for every kfunc call.

A by-value struct or union returned by a kfunc must be composed only of
scalars, since the verifier models the returned register bits as an unknown
scalar and a pointer field would otherwise be laundered into one, escaping
provenance and reference tracking.

A global subprogram must return a scalar in every return register. The
existing exemption for arena pointers now applies only when the return
value fits in R0 alone: both halves of a register pair carry a piece of a
>8 byte scalar, so an arena pointer in either of them is a leak rather than
a legitimate return value. A subprogram whose whole return value is an
arena pointer is unaffected.

A static subprogram is handled differently. The verifier walks into its
frame, so prepare_func_exit() propagates the return register(s) to the
caller. R0 holding a stack pointer has long been rejected outright there,
but R2 is deliberately not treated the same way. LLVM owns both sides of a
static call and is not bound by the ABI, so even with a 9..16 byte declared
return type it may leave R2 untouched when the caller only consumes the low
half; R2 can then hold an incidental stack pointer that is not a return
value at all, and rejecting the program would be a false positive.
Propagating the register as is would be worse: the callee frame is freed
immediately afterwards, leaving the caller with a PTR_TO_STACK that refers
to a frame which no longer exists. So the caller's R2 is marked
uninitialized instead, and only a caller that actually reads the returned
upper half fails. As with R0, a pointer into the caller's own frame is
scrubbed too, which is conservative but keeps the two registers consistent.

Once callers read R0:R2, an extension program can no longer replace a
function with a >8 byte return value: an extension's own return is
capped at 8 bytes by the program-exit check above, so it would leave R2
stale for the target's callers. btf_check_type_match() cannot catch
this, as it compares return types by btf_type->info only and an int
carries no vlen, so a 16-byte __int128 and an 8-byte long compare equal.
Reject such an attach in bpf_check_attach_target() instead.

  [1] llvm/llvm-project#190894
  [2] llvm/llvm-project#206876

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>

The R0:R2 return convention is derived from the BTF function prototype:
bpf_compute_subprog_ret_regs() inspects the return type of every
subprogram and records whether its value comes back in a register pair.

btf_check_subprog_call() can decide, at a call site, that this BTF is
not to be trusted and mark the subprogram unreliable, which happens when
compiler optimizations remove arguments from a static function or when a
mismatched type is passed to a global one. From that point on the
verifier falls back to conservative, R0-only, semantics for the
subprogram, while the compiled code keeps returning a pair and leaves
the upper half in R2 behind the verifier's back.

Rather than silently mistracking R2, reject a return value larger than
8 bytes as soon as the prototype it was derived from becomes unreliable.
Add subprog_ret_pair_unreliable() and test it at the two places that can
observe the flag: check_func_call(), for the call itself, and
prepare_func_exit(), for the return from an inlined static subprogram.

Note that the main program needs no such check: a >8 byte return from
subprog 0 is rejected at BPF_EXIT regardless of whether its BTF is
reliable. Callbacks need none either: a callback address only becomes a
PTR_TO_FUNC through check_ld_imm(), which already rejects any callback
returning more than 8 bytes.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>

Relax btf_distill_func_proto() to accept a by-value struct or union that
the R0:R2 convention added in earlier patches can carry:

 - a struct or union larger than 8 and up to 16 bytes, returned in the
   R0:R2 register pair, matching what LLVM emits for the BPF target;
 - a struct or union up to 8 bytes, returned in R0 alone.

A >8 byte scalar (__int128) was already accepted and is unchanged.
Everything else stays rejected: a return type larger than 16 bytes, and any
type that __get_type_size() cannot return in registers at all (e.g. an
array), which it already reports as ret < 0.

btf_distill_func_proto() also builds the trampoline (fentry/fexit/fmod_ret)
and struct_ops function models, so relaxing it widens what those can attach
to. A >8 byte return stays rejected on every path that reads the target's
return value: commit c48796a ("bpf: Reject >8 byte return values on
return-reading trampoline paths") covers fexit, fmod_ret and fsession plus
their _multi variants, and struct_ops, and an fentry-only trampoline never
sets BPF_TRAMP_F_CALL_ORIG so it does not touch the return value at all. A
struct or union of 8 bytes or less is newly accepted for those paths; its
single eightbyte is returned in R0 like any other scalar.

btf_validate_return_type() is relaxed as well, so that it accepts a
by-value struct or union up to 16 bytes in addition to void and scalars.

With btf_distill_func_proto() and btf_validate_return_type() relaxed, the
verifier, JIT, precision-backtracking and live-register support from the
earlier patches becomes reachable: this final patch enables <=16 byte
aggregate return values end to end.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>

Add selftests that exercise a 16-byte return value passed in the R0:R2
register pair, written in C so that they depend on the compiler lowering
the register-pair return. Covered are an __int128 return, a 16-byte struct
return (from a static and from a global subprogram) and a 16-byte union
return, plus __int128 and 16-byte struct returns from a kfunc, for which
bpf_kfunc_call_test_i128() and bpf_kfunc_call_test_ret_pair() are added to
bpf_testmod.

The R0:R2 convention is only emitted by LLVM 23 and newer. Each object
records in a read-only has_reg_pair_ret flag which compiler built it; where
that is false the programs are stubs and subtests report a skip rather than
a pass.

The two kfunc subtests further depend on the JIT: bpf_add_kfunc_call()
rejects a kfunc returning more than 8 bytes with -EOPNOTSUPP where
bpf_jit_supports_kfunc_ret_reg_pair() is false. Those calls therefore live
in an object of their own, and that load failing with -EOPNOTSUPP is what
turns the two subtests into skips, so no list of the JITs implementing the
pair needs to be kept here. A register-pair return from a BPF subprogram
needs no JIT support, so the remaining subtests run everywhere.

Both kfuncs are restricted to x86_64 and arm64, and the two subtests report
a skip elsewhere. pahole only BTF-encodes a function whose declared
arguments sit in the ABI's argument registers, and an architecture that
returns a value larger than 8 bytes through a hidden pointer (sret) shifts
every one of them by a register. s390x is such an architecture: there
pahole drops the function, resolve_btfids leaves the kfunc ID at 0, and
register_btf_kfunc_id_set() then fails at module init, so bpf_testmod does
not load at all and every test that needs it fails. This is not a property
of the compiler -- __SIZEOF_INT128__ is defined by gcc on s390x, and a
by-value struct return has nothing to do with __int128 in the first place --
so the guard is on the architecture. prog_tests/tracing_failure.c already
restricts a __int128 return the same way.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>

Add inline-asm tests, which do not depend on the compiler lowering a
register-pair return and so run regardless of the LLVM version, covering
what the C tests cannot reach. aggregate_ret_func.c exercises BPF-to-BPF
returns: a global subprogram whose R2 the caller may read, ones that leave
R2 uninitialised or holding a pointer, a static subprogram whose R2 stays
precise under backtracking, R2 liveness across a call, and a >8 byte return
at program exit. Six kfuncs returning aggregates by value are added to
bpf_testmod, and aggregate_ret_run.c calls them from inline asm to check
what comes back in R0:R2.

A negative arena test is added as well: a global subprogram with a
register-pair return that leaves an arena pointer in R2 is rejected, since
an arena pointer is only a valid return value when it is returned in R0
alone.

Three cases cover the boundaries of the new convention:

 - A return value larger than 16 bytes does not fit in R0:R2 and is
   rejected by btf_distill_func_proto(), ahead of the KF_FASTCALL and
   JIT-capability checks; one of the new kfuncs returns a 24-byte struct
   for this. The equivalent for a BPF subprogram cannot be written in C:
   from LLVM 23 on, a by-value return larger than 16 bytes is lowered to an
   sret pointer argument and the BTF the verifier reads says the function
   returns void, so the size bound in btf_validate_return_type() only
   guards hand-crafted BTF.

 - A static subprogram returning a struct that contains a pointer is
   accepted, and the caller can use the returned pointer. Unlike a global
   subprogram, whose caller models the return as an opaque scalar pair, a
   static one is verified inline, so prepare_func_exit() hands the caller
   real register state and the pointer stays tracked.

 - An extension cannot replace a function returning more than 8 bytes.
   btf_check_type_match() does not catch this, since it compares return
   types by btf_type->info alone and both an __int128 and a __u64 are
   BTF_KIND_INT with no vlen, so the rejection has to come from
   bpf_check_attach_target(). The test reuses the freplace failure harness
   in fexit_bpf2bpf.c, with aggregate_ret_target.c providing a target whose
   global subprogram returns in R0:R2.

The kfunc tests need the JIT to place the second half of a return value
into R2, which bpf_add_kfunc_call() only allows where
bpf_jit_supports_kfunc_ret_reg_pair() is true. In aggregate_ret_kfunc.c the
two tests that depend on getting past that check are tagged
__arch_x86_64/__arch_arm64/__arch_riscv64; the others are rejected earlier
(on KF_FASTCALL, on a >16 byte return, and on reading R2 after an 8-byte
struct return). In aggregate_ret_run.c the kfunc-calling programs are
dropped from the object when the load reports -EOPNOTSUPP and their
subtests are skipped, and the __int128 inline-asm test is split into a
BPF-to-BPF half, which needs no JIT capability and runs everywhere, and a
kfunc half.

Five of the six new kfuncs return a struct or union by value and take
arguments, so they join the x86_64/arm64 guard added in the previous patch;
see the comment there. That includes the one returning only 8 bytes: s390x
hands back a by-value aggregate through an sret pointer whatever its size,
so its declared arguments are displaced just the same and pahole skips the
function. The sixth returns 24 bytes but takes no argument, leaving nothing
for the sret pointer to displace, and pahole does encode it there.
Both objects calling the guarded kfuncs reference every one of them from
__kfunc_btf_root(), so neither can load at all where those kfuncs are
absent; the aggregate_ret_kfunc tests and the four aggregate_ret_run
subtests are skipped as a group elsewhere. aggregate_ret_func.c calls no
kfunc and keeps running everywhere.

R0 holds bytes 0..7 of a return value and R2 bytes 8..15, so where a member
sits inside a register depends on the endianness of the target. The checks
in aggregate_ret_run.c that read a member out of half a register are built
for a little-endian target only; arm64 is the one JIT implementing the pair
that can be built big endian, and that configuration is left for later.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>

Add two __failure tests covering the callback return-size checks:

 - timer_ret_pair_fail: a bpf_timer callback declared to return more than
   8 bytes, rejected by check_ld_imm() where the callback's PTR_TO_FUNC is
   created, with "callback function with >8-byte return value is not
   supported".

 - exceptions_ret_pair_fail: an exception callback declared to return more
   than 8 bytes, rejected by do_check_common() when the callback
   subprogram is verified, with "exception cb cannot return value larger
   than 8 bytes".

Both callback bodies are written in inline asm so that the tests do not
depend on LLVM 23 R0:R2 codegen and run on any compiler. The verifier reads
the return type from BTF rather than from the instructions, so the >8 byte
return prototype is supplied through __btf_func_path(), pointing at a
companion btf__*.c program that exists only to carry that BTF.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>

kfuncs may now return a value larger than 8 bytes and up to 16 bytes (a
scalar-only struct or union, or an __int128), passed back in the R0:R2
register pair. Add a kfunc return-value section documenting this,
including that a struct or union up to 8 bytes is returned in R0 alone,
which struct and union members are accepted, that the R0:R2 register pair
requires JIT support (bpf_jit_supports_kfunc_ret_reg_pair()), and that a
return value larger than 16 bytes is unsupported.

Also note that the same convention applies to BPF subprogram returns, and
document the consequence for a global subprogram: it must assign both
halves of a register-pair return, since an unassigned R2 may be left
holding a pointer argument and is then rejected as a leak.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>